1. Field of the Invention
This invention generally relates to the field of electronic music and audio signal processing and, particularly, to a digital audio signal processing technique for providing timbral change in arbitrary audio input signals and stored complex, dynamically controlled, time-varying digital signals as a function of the amplitude of the signal being processed.
2. Description of the Prior Art
In the field of electronic music and audio recording it has long been an ambition to achieve two goals: Music that is synthesized or recorded with maximum realism and music that selectively includes special sounds and effects created by electronic and studio techniques. To achieve these goals, electronic musical instruments for imitating acoustic instruments (realism) and creating new sounds (effects) have proliferated. Signal processors have been developed to make these electronic instruments and recordings of any instruments sound more convincing and to extend the spectral vocabularies of these instruments and recordings.
While considerable headway has been made in various synthesis techniques, including analog synthesis using oscillators, filters, etc., and frequency modulation synthesis, the greatest realism has been attained by the technique of digitally recording small segments of sound, colloquially known as samples, into a digital signal memory for playback by a keyboard or other controller. This sampling technique yields some very realistic sounds. However, sampling has one very significant drawback: Unlike acoustic phenomena, the timbre of the sound is the same at all playback amplitudes. This results in uninteresting sounds that are less complex, controllable and expressive than the acoustic instruments they imitate. Similar problems occur to different degrees with synthesis techniques.
To increase the realism of synthesized music, a number of signal processing techniques have been employed. Most of these processes, such as reverberation, were originally developed for the alteration of acoustic sounds during the recording process. When applied to synthesized waveforms, they helped increase the sonic complexity and made them more natural sounding. However, none of the existing devices are able to relate timbral variation to changes in loudness with any flexibility. This relationship is well understood to be critical to the accurate emulation of acoustic phenomena. This invention provides a means of relating these two parameters, the processed result being more realistic and interesting than the unprocessed signal which has the same timbre at all input amplitudes.
A number of signal processing techniques have been developed for achieving greater variety, control and special effects in the sound generating and recording process. In addition to the realism mentioned above, these signal processors have sought to extend the spectrum of available sounds in interesting ways. Also, to a large extent many of the dynamic techniques of signal processing have been well investigated for special effects, including time/amplitude, time/frequency, and input/output amplitude. These processes include, reverberators, filters, compressors and so on. None of these devices have the property of relating the amplitude of the input to the timbre of the output in such a way as to add musically useful and controllable harmonics to the signal being processed.
There are three areas of prior art that have direct bearing upon the invention: (1) The use of non-linear transformation in non-real-time mainframe computer synthesis, (2) the use of non-linear transformation in real-time sine-wave based hardware additive synthesis, and (3) the generation of new samples by using pre-existing samples as a non-dynamic input to a non-linear transformation means. Non-linear transformation of audio for music synthesis, also known as waveshaping, via the use of look-up tables has been in common use in universities worldwide since the mid-1970's. The seminal work in this field was done by Marc LeBrun and Daniel Arfib and published in the Journal of the Audio Engineering Society, V. 27, No. 4 and V. 27 No. 10. The work described in these writings gives an overview of waveshaping and makes extensive use of Chebyshev polynominals. The work done in this area consists primarily of the distortion of sine waves in order to achieve new timbres in music synthesis. There was a particular focus on brass instrumental sounds, as evidenced by the work of James Beauchamp, (Computer Music Journal V. 3 No. 3 Sept. 3, 1979) and others.
Hardware synthesis exploiting the non-linearity of analog components has been employed in music to distort waveforms for many years. Research in this area was done by Richard Schaefer in 1970 and 1971 and published in the Journal of the Audio Engineering Society, V. 18, No. 4 and V. 19, No. 7. In this literature he discusses the equations employed to achieve predictable harmonic results when synthesizing sound. With a sine wave input and using Chebyshev polynomials to determine the non-linear components used on the output circuitry, different waveforms were synthesized for electronic organs. More recently, Ralph Deutsch has employed hardware lookup tables as a real-time variation of the earlier mainframe synthesis techniques (U.S. Pat. Nos. 4,300,432 and 4,273,018). The Deutsch patents differ from the work by LeBrun, Arfib et al only inasmuch as multiple sine waves, orthogonal functions, or piecewise linear functions rather than single sine waves are input into the look-up table to achieve the synthesis of the desired output.
One limitation of the above mentioned uses of non-linear transformation are their employment in synthesis environments that did not allow real-time arbitrary audio input. By embedding the look-up tables or non-linear analog components in the synthesis circuitry or software, distortion of audio signals coming from outside the synthesis system was rendered impossible.
One advantage of this invention lies in its capacity to accept and transform arbitrary real-time audio input or a stream of digital signals which is representative of such audio input. This opens up the possibility of performing non-linear transformation upon acoustic signals. Also, original or modified audio signals produced by any synthesis technique can be processed by a waveshaper. It also enables the insertion of the waveshaping circuitry into various signal processor configurations. Thus, it can be included as part of the recording/mixdown process before or after other signal processors, such as compressors, reverberators and filters.
The first two techniques described both possess another limitation in that they describe tone generators based on additive synthesis of sine or other elementary functions. The signals to be transformed are static, computed, periodic waveforms which are processed to add time varying timbral qualities. These computed-function based inputs comprise a limited class of periodic waveforms and hence produce a narrow range of sonic qualities. The more interesting case of devices which include digital signal memories (e.g. samplers) for storing complex, time-varying audio data is not addressed or implied in either of these techniques.
While some of the prior art employs memory to store signals to be transformed, these devices store periodic, elementary functions (e.g. sine waves). It is possible to calculate the values of these functions from point to point in hardware but it is simpler and more economical to store pre-computed functions in memory. This art does not exploit the fundamental property of memory to store arbitrary complex, time-varying signals.
When these complex, time-varying stored digital waveforms are non-linearly transformed, a new class of musically useful timbres is produced. Since the digital signal memory can store essentially arbitrary audio signals, the operation of the transform memory is identical to that described above for arbitrary input with the added advantage that sonic events can be conveniently stored, selected, triggered and controlled, as is the case with today's conventional samplers.
There are several advantages to including the transformation memory within an architecture that includes a digital signal memory, such as a sampler. One advantage is that a single transform memory can be applied to multiple notes and/or waveforms through time-multiplexing of the table. This eliminates the undesirable mixing effects that occur when multiple notes are non-linearly processed. It is also possible to eliminate mixing by dedicating a separate physical transform memory to each active note, but this approach is inherently more costly than multiplexing a single memory. A further advantage of the invention is that the addition of a transform memory provides a means for economically extending the available set of sounds by applying various timbral modifications to each of the original sounds. Thus, for example, a set of 16 sampled sounds may provide 48 different sounds with the addition of two very different transform memories--the original 16 plus 16 of each transformed set.
The third technique described above, that of generating new samples by using pre-existing samples as a non-dynamic input to a non-linear transformation means, has been implemented in a software product called Turbosynth by the Digidesign Company. Turbosynth is designed to create new samples for musical use by using one or more of several techniques. These include synthesizing sounds and processing pre-existing samples and synthesized waveforms with a number of different tools, such as volume envelopes, mixers, filters, etc., which are executed in software on a Macintosh computer. Pertinent to this invention, non-linear transformation, or waveshaping, is one of the tools included. Turbosynth is typically used to create new samples which are then exported to the memory of a sampling synthesizer for performance.
By using the waveshaping tool in Turbosynth, distortion of arbitrary audio input is possible in as far as the arbitrary audio input is not real-time and is static with regard to any external control parameters. Only samples, or finite segments of stored digital audio, may be processed. Although the waveform of the sample may vary in time, unless it or some other aspect of the architecture is recalculated, none of its parameters vary; the data input to the waveshaper is always exactly the same. The waveshaping operation(s) is/are applied to the waveform only once, not continuously. It is thus limited in that dynamic timbral variation as a function of real-time parameters such as key velocity, cannot be achieved. It is possible to dynamically vary the amplitude and other parameters of the sample playback after the sample has been exported to the sampling synthesizer. However, at this point, the waveshaping process has been completed and the dynamic changes have no effect upon the timbre of the sound.
To accelerate the recalculation process, Digidesign offers a hardware product called the Sound Accelerator. With this device, it is possible to preview the changes made to a sound created in Turbosynth in real time by playing notes on a music keyboard attached to the Macintosh. However, while different pitches may be input to the waveshaper, no other dynamic parameter variations can be affected. The waveshaper is thus used as a tool for generating new, fixed timbres and not, like the present invention, as a processor for achieving dynamic timbral variation.
Structurally, Turbosynth, as it may relate to the present invention, can be thought of as shown in FIG. 20. In this example, only the waveshaper tool is employed. A digital audio sample from a sampler 200 is transferred to digital signal memory file 130a in the Macintosh computer 201. It is then processed via the waveshaper tool, which is a look up table 103. The output of the look up table is a second digital signal memory file 130b which may optionally be previewed using the Macintosh D/A converter 104 and speaker 125. If the user wishes to use the sound for performance, it would be transferred back to the sampler 200. The transformed sound is now fixed in the sampler's memory and when the instrument is played, all RMS amplitude changes, filter changes, and son on, are performed upon the new, fixed timbre.
The crucial limitation of this structure is that it places the look up table prior to the performance control mechanism of the sampler. As described above, this precludes the most powerful aspect of waveshaping, i.e. its ability to produce not one new timbre but a continuum of new timbres as a function of input amplitude.